Dashboard Overview

This dashboard compares coaching bot versions based on session metrics, annotations, and user engagement.

Generated: 2025-11-10 15:07:32

Dashboard Filters

to
Enter one participant ID per line. Leave empty to show all participants.
Note: Date and participant filters work for the Session Volume chart. Refrigerator filter applies to all graphs and tables. Participant filter for tables requires dashboard regeneration. Outlier filtering affects the progression line graph and median words/messages tables.

Summary Metrics by Version

Coaching Bot Version # Sessions # Annotated Sessions Refrigeration Examples (%) Median Human Words per Session Average Session Rating
Control bot 4231 0 0.0% 37.0 4.80
Coaching bot V3 888 80 0.0% 124.0 4.71
Coaching bot V4 949 172 33.1% 105.0 4.85
Coaching bot V5 1291 62 37.1% 152.0 4.53
Coaching bot V6 2168 10 70.0% 120.5 4.65
Total (All Versions) 9527 324 26.5% 120.5 4.71

Summary Metrics - All Versions vs Refrigerator Examples

Category # Sessions # Annotated Sessions Refrigeration Examples (%) Median Human Words per Session Average Session Rating
All Versions 9527 324 26.5% 120.5 4.71
Refrigerator Example Sessions Only 87 87 100.0% 154.0 4.81

Refrigerator Example Rate by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario--100.0%100.0%-100.0%
Microlearning--72.3%100.0%-86.2%
Microlearning vaccines---66.7%100.0%83.3%
Motivational interviewing--62.5%100.0%100.0%87.5%
Visit check in--100.0%100.0%-100.0%
Unknown------
Total (All Methods)--83.7%93.3%100.0%91.4%

Average FLW Score by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario-4.774.774.464.634.66
Microlearning--4.854.554.634.68
Microlearning vaccines---4.474.654.56
Motivational interviewing--4.874.694.674.74
Visit check in-4.634.944.644.664.72
Unknown4.80---4.864.83
Total (All Methods)4.804.704.864.564.684.70
Rating Collection Statistics:
• Rating Questions: 86.3% of sessions (8250 out of 9557) contain rating questions
• Actual Ratings: 84.3% of sessions (8058 out of 9557) have extractable ratings

Median Number of Participant Messages per Session by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario-12.011.014.012.012.0
Microlearning--13.014.513.013.0
Microlearning vaccines---13.013.013.0
Motivational interviewing--16.019.017.017.0
Visit check in-13.015.018.017.016.0
Unknown9.03.03.0-12.06.0
Total (All Methods)9.012.013.014.513.013.0

Median User Words per Session by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario-127.0100.0151.0127.0127.0
Microlearning--87.0122.090.090.0
Microlearning vaccines---93.590.091.8
Motivational interviewing--108.0167.5127.0127.0
Visit check in-118.5130.0209.5169.5149.8
Unknown37.08.013.0-54.025.0
Total (All Methods)37.0118.5100.0151.0108.5109.4

Median User Words - Session Progression Analysis

Volume of Session per Coach Version

Session Count by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario16372585534471896
Microlearning--247186412845
Microlearning vaccines---176410586
Motivational interviewing--219190437846
Visit check in62422191864481101
Unknown422496-144253
Total (All Methods)4231888949129121689527

Definitions

Version Definitions

This dashboard compares five bot categories based on experiment IDs and version ranges:

  • Control bot: Experiment ID: 1027993a-40c9-4484-a5fb-5c7e034dadcd (All versions)
  • Coaching bot V3: Experiment ID: e2b4855f-8550-47ff-87d2-d92018676ff3 (All versions)
  • Coaching bot V4: Experiment ID: b7621271-da98-459f-9f9b-f68335d09ad4 (Version 13 and above)
  • Coaching bot V5: Experiment ID: 5d8be75e-03ff-4e3a-ab6a-e0aff6580986 (Version 1 to 4)
  • Coaching bot V6: Experiment ID: 5d8be75e-03ff-4e3a-ab6a-e0aff6580986 (Version 5 and above)
Coaching Methods

Coaching methods are detected through a 3-tier approach:

  • Scenario: Roleplay and scenario-based coaching
  • Microlearning: Quiz and microlearning sessions
  • Microlearning vaccines: Vaccine-specific quiz sessions
  • Motivational interviewing: Motivational interview techniques
  • Visit check in: Home visit debrief sessions
Average FLW Score by Method and Version

This metric shows the average session rating (1-5 scale) grouped by coaching method and bot version.

  • Calculation: Average of user ratings for sessions ending with "How useful did you find this coaching session? Please rate it from 1 to 5"
  • Rating Scale: 1 (not useful) to 5 (very useful)
  • Method Detection: Based on session tags (coach_method_*) or message content analysis
  • Version Detection: Based on experiment ID and version tags from last message
  • Purpose: Identify whether coaching methods receive higher ratings with bot evolution
Median Number of Participant Messages per Session by Method and Version

This metric shows the median number of messages participants send per session, grouped by coaching method and bot version.

  • Calculation: Median count of user messages within a session
  • Method Detection: Based on session tags (coach_method_*) or message content analysis
  • Version Detection: Based on experiment ID and version tags from last message
  • Message Count: Total number of user messages in a session
Median User Words per Session by Method and Version

This metric shows the median number of words users type per session, grouped by coaching method and bot version.

  • Calculation: Median word count of all user messages within a session
  • Method Detection: Based on session tags (coach_method_*) or message content analysis
  • Version Detection: Based on experiment ID and version tags from last message
  • Word Count: Total words across all user messages in a session
  • Outlier Filtering: Optional checkbox to exclude sessions with >50 messages or >1000 words
Session Numbering:

For line graph analysis, sessions are numbered chronologically per participant:

  • Participant ID: Based on participant.identifier field
  • Chronological Order: Sorted by session created_at timestamp
  • Session Number: Position in participant's session sequence (1st, 2nd, 3rd, etc.)